Skip to content

wasm: in-browser OCR via ONNX Runtime Web interop (#157 stage 1) - #158

Merged
artiz merged 40 commits into
docling-project:masterfrom
artiz:claude/wasm-ml-stage1
Jul 25, 2026
Merged

wasm: in-browser OCR via ONNX Runtime Web interop (#157 stage 1)#158
artiz merged 40 commits into
docling-project:masterfrom
artiz:claude/wasm-ml-stage1

Conversation

@artiz

@artiz artiz commented Jul 24, 2026

Copy link
Copy Markdown
Contributor

First slice of the browser ML pipeline: scanned images OCR fully client-side.

docling-pdf grows an ocr-prep feature (pure Rust + the image crate, no ort/pdfium): the ONNX-free half of the PP-OCRv3 recognition pipeline — line segmentation, crop prep, deterministic width-batching, CTC decode, dictionary handling — moved verbatim from ocr.rs into a public ocr_prep module. ocr.rs now consumes it, so there is exactly ONE implementation of the pre/post-processing; the native path is byte-identical after the refactor (verified: the scanned-tiff conversion diffs empty against the pre-refactor output, full docling-pdf suite green). Any wasm-vs-native drift can therefore only come from the runtime's kernels — the #157 conformance story depends on that property.

docling-wasm (default ocr feature): ocr_image(bytes, dict, session, to) — decode the image in Rust, segment/prep/batch via ocr_prep, delegate each width-batch to a JS RecSession wrapper around an ort-web InferenceSession (run(n, h, w, data) -> {data, dims}), CTC- decode, and export a DoclingDocument as Markdown or docling JSON. No layout model yet (stage 2): the whole image is one text region split by the ink-projection profile — right for single-column scans, degraded on complex layouts, and said so in the docs.

www/ocr.html: complete drop-an-image demo — ort-web from CDN, the ~10 MB en_PP-OCRv3 model + dictionary fetched from their public hosting and browser-cached, timing readout. Compiles for wasm32-unknown-unknown and the host (rlib tests); clippy clean; the existing wasm CI check is unaffected.

Closes #157

Claude-Session: https://claude.ai/code/session_01EY5KAiquN4YpVf2PXEQkVT

artiz and others added 30 commits July 25, 2026 09:54
…stage 1)

First slice of the browser ML pipeline: scanned images OCR fully
client-side.

docling-pdf grows an `ocr-prep` feature (pure Rust + the image crate, no
ort/pdfium): the ONNX-free half of the PP-OCRv3 recognition pipeline —
line segmentation, crop prep, deterministic width-batching, CTC decode,
dictionary handling — moved verbatim from ocr.rs into a public ocr_prep
module. ocr.rs now consumes it, so there is exactly ONE implementation
of the pre/post-processing; the native path is byte-identical after the
refactor (verified: the scanned-tiff conversion diffs empty against the
pre-refactor output, full docling-pdf suite green). Any wasm-vs-native
drift can therefore only come from the runtime's kernels — the docling-project#157
conformance story depends on that property.

docling-wasm (default `ocr` feature): `ocr_image(bytes, dict, session,
to)` — decode the image in Rust, segment/prep/batch via ocr_prep,
delegate each width-batch to a JS `RecSession` wrapper around an
ort-web InferenceSession (`run(n, h, w, data) -> {data, dims}`), CTC-
decode, and export a DoclingDocument as Markdown or docling JSON. No
layout model yet (stage 2): the whole image is one text region split by
the ink-projection profile — right for single-column scans, degraded on
complex layouts, and said so in the docs.

www/ocr.html: complete drop-an-image demo — ort-web from CDN, the ~10 MB
en_PP-OCRv3 model + dictionary fetched from their public hosting and
browser-cached, timing readout. Compiles for wasm32-unknown-unknown and
the host (rlib tests); clippy clean; the existing wasm CI check is
unaffected.

Refs docling-project#157

Signed-off-by: artiz <artem.kustikov@gmail.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EY5KAiquN4YpVf2PXEQkVT
… stage 2)

Scanned PDFs and images now convert fully client-side through the real
pipeline shape: RT-DETR layout detection + PP-OCRv3 recognition via
ONNX Runtime Web, everything else in Rust — the native pipeline's own
functions, no second implementation.

docling-pdf (all under the existing ocr-prep feature, wasm-safe):
- layout.rs: input packing (640x640 CHW) and RT-DETR decoding (sigmoid
  top-q, center->corners, per-label thresholds downstream) extracted as
  layout_input/decode_layout; the native batch path now decodes through
  the same function per page.
- ocr_prep: prep_region_lines — the exact region-crop/line-split/prep
  gathering native ocr_page does — moved out and shared; ocr.rs
  consumes it.
- new scanned module: the Worker::process refinement chain
  (label thresholds -> resolve -> orphans -> false-picture drops ->
  contained drops), page assembly with the geometric table fallback
  (the lite profile: TableFormer is stage 3), and cross-page
  continuation merging, re-exposed without the ml feature.
Native path byte-identical after the extraction — the scanned-tiff
conversion diffs empty against the pre-refactor output (rebuilt and
re-verified after a disk-full broke the first attempt), full
docling-pdf/docling suites green.

docling-wasm:
- ScannedConverter: feed pdf.js-rendered RGBA pages ({scale: 2} — the
  native RENDER_SCALE) one by one, finish() assembles the document with
  the same continuation merging as native; page_count() for progress.
- convert_scanned_image: one-shot image path (its own page at scale 1,
  like native).
- LayoutSession interop mirrors RecSession: run(data) ->
  {logits: {data, dims}, boxes: {data, dims}}.
- www/scan.html: complete demo — pdf.js from CDN, layout int8
  (~165 MB, models-v1 release, browser-cached) + rec model + dict,
  per-page progress, timing.

Stage 3 (TableFormer + enrichment in the browser) remains: the
autoregressive decoder loop is deeply ort-typed and needs its own
careful extraction pass; table regions meanwhile take the geometric
reconstruction, matching --no-table-former.

Refs docling-project#157

Signed-off-by: artiz <artem.kustikov@gmail.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EY5KAiquN4YpVf2PXEQkVT
… field

The workspace-wide clippy job unifies features: docling-cli enables
docling-pdf/ml alongside docling-wasm, so PdfPage grows its ml-gated
`image` field and docling-wasm's struct literal stopped compiling
(E0063). A cfg-aware constructor (from_cells, under ocr-prep) spells
the field only when it exists; the browser pipeline builds its pages
through it. Verified with the CI command verbatim
(cargo clippy --workspace --all-targets -- -D warnings) plus the
wasm32 target check.

Refs docling-project#157

Signed-off-by: artiz <artem.kustikov@gmail.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EY5KAiquN4YpVf2PXEQkVT
)

First real-browser test fed ocr.html a dark-theme screenshot and got
"(no text found)": the whole scan pipeline assumes dark ink on light
paper — the line segmentation thresholds pixels *darker* than the page
mean, so light-on-dark input yields garbage. New
ocr_prep::normalize_polarity inverts a predominantly dark page (mean
luma below mid-gray) before segmentation and inference; both browser
entry points (ocr_image, ScannedConverter::add_page — layout assumes
scan polarity too) run it. The native pipeline deliberately does NOT:
its input is scanned paper and the conformance baseline stays pinned.

Test: the inverted synthetic page misfires raw and recovers exactly
after normalization; a light page passes through untouched.

Refs docling-project#157

Signed-off-by: artiz <artem.kustikov@gmail.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EY5KAiquN4YpVf2PXEQkVT
The demo pages hardcoded the English PP-OCRv3 model, so any non-Latin
scan (e.g. Cyrillic) decoded to mojibake through the wrong dictionary.
The wasm API is language-agnostic — model and dict are injected from
JS — so the fix is demo-side only: both ocr.html and scan.html get a
recognition-language <select> (en / cyrillic / ch) backed by a lazy
per-language ort.InferenceSession cache; models load on first use and
stay browser-cached (~10 MB each).

Refs docling-project#157

Signed-off-by: artiz <artem.kustikov@gmail.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EY5KAiquN4YpVf2PXEQkVT
SWHL/RapidOCR ships no cyrillic PP-OCRv3 export — the previous URL
404'd and ort-web choked on the HTML error page ("protobuf parsing
failed"). Point the cyrillic entry at the
cycloneboy/cyrillic_PP-OCRv3_rec_infer HF repo instead, which carries
both the ONNX export and its paired 163-char dictionary (verified:
output tensor is [n, t, 165] = blank + dict + space, matching
dict_chars' table layout).

Refs docling-project#157

Signed-off-by: artiz <artem.kustikov@gmail.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EY5KAiquN4YpVf2PXEQkVT
Without the fetch-images feature (the wasm32 build) the fetch_remote
stub had no caller left — fetch_cached, its only user, is feature-gated
— and FsImageResolver::base_url is stored but only read by the gated
remote-URL resolution. Drop the stub and allow(dead_code) the field so
`wasm-pack build` stays warning-free without forking new()'s signature
per feature shape.

Refs docling-project#157

Signed-off-by: artiz <artem.kustikov@gmail.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EY5KAiquN4YpVf2PXEQkVT
The v3 cyrillic export reads Cyrillic but drops spaces and flips case
on lookalike glyphs. Side-by-side on a rendered test line (same
preprocessing as prep_line, greedy CTC): v3 gives "еш: pаs" with glued
words, PaddlePaddle/cyrillic_PP-OCRv5_mobile_rec_onnx decodes the whole
line exactly, spaces included. Same interop contract — input
[n,3,48,W], output [n,t,852] = blank + 850 dict chars + space, matching
dict_chars' table.

The v5 dictionary only exists inside the repo's inference.yml, so a
flattened cyrillic_v5_dict.txt ships next to the demo pages.

Refs docling-project#157

Signed-off-by: artiz <artem.kustikov@gmail.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EY5KAiquN4YpVf2PXEQkVT
The paddle2onnx v5 export carries unused initializers, and ort logs a
console warning for each one it prunes at session load. Harmless but
noisy — cap logSeverityLevel at errors for all demo sessions.

Refs docling-project#157

Signed-off-by: artiz <artem.kustikov@gmail.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EY5KAiquN4YpVf2PXEQkVT
While a model loads or a document converts, the demos now disable the
upload controls (drop zone, file input, language selector), pulse the
drop-zone border and spin the status line, so a click mid-run can't
start a second conversion. Model downloads stream through a
fetch-with-progress helper showing a live "x / y MB" line — the 165 MB
layout model no longer sits on a static message for minutes on a first
visit. Switching language now warms the model immediately instead of on
the next drop.

Also cap ort logSeverityLevel at errors: the paddle2onnx v5 export
carries unused initializers and ort logged a console warning for each
one it pruned at session load.

Refs docling-project#157

Signed-off-by: artiz <artem.kustikov@gmail.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EY5KAiquN4YpVf2PXEQkVT
The demo pointed at layout_heron_int8.onnx, but the release only hosts
the fp32 layout_heron.onnx (172 MB) — the int8 variant is produced
locally by scripts/install/quantize_models.py and was never uploaded,
so the page died on a 404 before ever becoming ready.

Refs docling-project#157

Signed-off-by: artiz <artem.kustikov@gmail.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EY5KAiquN4YpVf2PXEQkVT
…n't CORS-fetchable)

GitHub Release assets are served from release-assets.githubusercontent.com
with no Access-Control-Allow-Origin header, so a browser fetch of the
layout model failed with a CORS error the moment the URL stopped 404ing.
The recognition models avoid this only because Hugging Face sends CORS
headers; the official docling-layout-heron-onnx on HF can't substitute —
it exports a different contract (images + orig_target_sizes in,
labels/boxes/scores out) than our decode_layout expects (raw
logits/pred_boxes).

So load the layout model from the page's own origin: prefer
./layout_heron_int8.onnx (~68 MB), fall back to ./layout_heron.onnx
(~172 MB), and if neither is present open a setup note explaining how to
fetch them (download_dependencies.sh) and drop one next to the page.
These are the exact conformance-validated exports the native pipeline
uses; *.onnx is already gitignored so the copied model isn't committed.

Refs docling-project#157

Signed-off-by: artiz <artem.kustikov@gmail.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EY5KAiquN4YpVf2PXEQkVT
….sh in www/

Point the same-origin layout paths at ./models/ and document running the
installer from crates/docling-wasm/www/ — it writes ./models/ relative to
the current directory, so the page reads ./models/layout_heron_int8.onnx
(fallback ./models/layout_heron.onnx) with no copy step. gitignore the
www/models/ tree the installer populates.

Refs docling-project#157

Signed-off-by: artiz <artem.kustikov@gmail.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EY5KAiquN4YpVf2PXEQkVT
Show a running "Ns/page avg" in the status line as pages convert, log
each page's wall time to the console, and print a total + average when
the run finishes — so a slow multi-page PDF is measurable instead of
just "page 2/10 …".

Refs docling-project#157

Signed-off-by: artiz <artem.kustikov@gmail.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EY5KAiquN4YpVf2PXEQkVT
The release now hosts layout_heron_int8.onnx, so a fresh
download_dependencies.sh drops it under ./models/ and the page already
prefers it. Report which export actually loaded ("ready (int8 layout)")
so the ~2.4x-faster path is visible instead of guessed.

Refs docling-project#157

Signed-off-by: artiz <artem.kustikov@gmail.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EY5KAiquN4YpVf2PXEQkVT
…orker

ORT Web ran single-threaded because SharedArrayBuffer needs the page to
be cross-origin isolated, which needs COOP+COEP response headers a static
file server can't set. coi.js is a service worker that re-serves every
response with COOP: same-origin + COEP: credentialless (credentialless so
the jsDelivr and Hugging Face cross-origin fetches still load without
their own CORP header); it reloads once on first visit to take control.
With isolation in place both demos set ort.env.wasm.numThreads to the
core count (capped at 8); browsers without support stay single-threaded
and still work. SIMD is auto-selected by the ORT bundle.

Refs docling-project#157

Signed-off-by: artiz <artem.kustikov@gmail.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EY5KAiquN4YpVf2PXEQkVT
Rasterize page p+1 while page p is being converted. pdf.js raster is
main-thread and the ORT layout/OCR inference runs on worker threads (once
cross-origin isolated), so the render of the next page now overlaps the
convert of the current one instead of running strictly after it.
getImageData hands back an independent buffer, so a single canvas
ping-pongs across pages safely. Per-page timing now reports convert time
plus an honest wall-clock total that reflects the overlap.

Refs docling-project#157

Signed-off-by: artiz <artem.kustikov@gmail.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EY5KAiquN4YpVf2PXEQkVT
Even with multi-threaded ORT, the main thread still froze during a
conversion: pdf.js rasterizes on it, and the Rust pre/post-processing
inside add_page (line segmentation, CTC decode, reading-order assembly)
runs synchronously on the calling thread. Move the whole pipeline off the
main thread so the page stays interactive.

Factor the DOM-free pipeline (model loading, rasterization, OCR/layout)
into pipeline.js — parameterized only by an onStatus reporter and a
makeCanvas factory — and drive it from worker.js, which rasterizes into
an OffscreenCanvas and posts status/result back. scan.html becomes a thin
UI: it spawns the worker, relays progress to the status line, and
transfers the file bytes in. If module workers or OffscreenCanvas are
unavailable it falls back to running pipeline.js on the main thread
(functionally identical, just blocking); a worker-side convert failure
retries once on the main thread so a result is always produced.

Also folds in an ORT warm-up (one blank layout+rec inference at boot) so
the first real page no longer pays the lazy kernel/thread init.

Refs docling-project#157

Signed-off-by: artiz <artem.kustikov@gmail.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EY5KAiquN4YpVf2PXEQkVT
Running pdf.js inside the Web Worker fell back to its "fake worker" mode
(a worker can't reliably spawn pdf.js's nested worker), which garbled the
rasterized line crops: layout still found the page structure (it downsizes
the whole page to 640x640) but recognition decoded noise.

Split the two: pdf.js rasterization runs on the main thread again — a real
HTMLCanvas, correct pixels, pdf.js's own worker — and only the heavy wasm
work (layout + OCR inference and the synchronous Rust pre/post-processing
in add_page, which is what actually froze the UI) runs in the worker.
Ready RGBA buffers transfer to the worker per page; rasterizing page p+1
overlaps converting page p. The worker no longer imports pdf.js and needs
no OffscreenCanvas, so the only requirement for the fast path is module
Worker support; without it the OCR runs same-thread (identical output,
blocking). pipeline.js is now the wasm-only OCR module with a document
lifecycle (startDoc/addPage/finishDoc) plus convertImage.

Refs docling-project#157

Signed-off-by: artiz <artem.kustikov@gmail.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EY5KAiquN4YpVf2PXEQkVT
…r ORT calls)

The exact-width grouping runs one recognition call per distinct line
width — on a text-dense page (most lines differing) that is nearly one
ORT call per line, and every call carries thread-sync overhead under ORT
Web. Add width_batches_padded / batch_input_padded: group lines into
REC_BATCH chunks sorted by width and pad each to the batch max. The
padding is the input buffer's zero value — mid-gray in the model's
[-1, 1] space, exactly what PP-OCRv3 trains/serves with in batched
inference — so the decode is output-equivalent to one-at-a-time
recognition (measured: 0 diffs on a mixed Latin+Cyrillic sample). A
1.5x width-ratio guard bounds padding waste. Wire the two wasm callers
(ocr.rs, scanned.rs) to the padded path; the native pipeline keeps the
bit-identical exact-width width_batches, so the byte-for-byte
conformance baseline is untouched.

Refs docling-project#157

Signed-off-by: artiz <artem.kustikov@gmail.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EY5KAiquN4YpVf2PXEQkVT
…er (fewer ORT calls)"

This reverts commit 1c871c7. Padding every line in a batch up to the
batch's max width inflates recognition compute (a narrow line padded to
a wide one does several times the conv work), and on real pages line
widths vary enough that the padding waste outweighs the saved per-call
overhead — measured ~9.6-10.4s/page vs ~8.0s/page with the exact-width
grouping. With multi-threaded ORT the per-run overhead the batching
targeted is already cheap, so the trade doesn't pay. Back to the
bit-identical exact-width batches.

Refs docling-project#157

Signed-off-by: artiz <artem.kustikov@gmail.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EY5KAiquN4YpVf2PXEQkVT
wasm-pack warns when license.workspace resolves but no LICENSE sits next
to the crate. Copy the workspace MIT license into the crate directory.

Signed-off-by: artiz <artem.kustikov@gmail.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EY5KAiquN4YpVf2PXEQkVT
…ore into tf_core

Groundwork for TableFormer in the browser: move the ONNX-free half of the
pipeline — the 448 encoder-input preprocessing, the autoregressive loop's
structure corrections (xcel→lcel, lcel-after-ucel→fcel) and bbox
bookkeeping (tag_H_buf / bboxes_to_merge), span merging, and the OTSL→grid
layout — into a new ocr-prep-gated tf_core module. The wasm build will run
these exact functions and delegate only the three ONNX graphs to ONNX
Runtime Web, the same one-implementation approach that keeps the browser
OCR/layout output matching native.

Native tableformer.rs (ml feature) now delegates to tf_core: preprocess
wraps tf_core::preprocess_input in an ort tensor; predict_otsl uses
tf_core::correct; predict_table_structure drives tf_core::BboxBook. Pure
code-motion — the owned-value KV-cache fast path and every ort call are
untouched, so native output is byte-identical. Unit tests cover the moved
pure functions (corrections, bbox bookkeeping, span build, argmax — noting
argmax ties to the last index via Rust max_by, kept verbatim).

Refs docling-project#157

Signed-off-by: artiz <artem.kustikov@gmail.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EY5KAiquN4YpVf2PXEQkVT
Scanned PDFs frequently store pages as JPEG2000 (or JBIG2) images, which
pdf.js v6 decodes with a separate wasm module; without a wasmUrl it warns
"Ensure that the wasmUrl API parameter is provided" and drops the image,
so the rasterized page — and the OCR run on it — comes out blank. Pin
pdf.js to 6.1.200 so the module, worker, and asset URLs all match, and
pass wasmUrl (plus cMapUrl for CJK and standardFontDataUrl) to
getDocument. wasm/ and cmaps/ verified present on the CDN.

Refs docling-project#157

Signed-off-by: artiz <artem.kustikov@gmail.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EY5KAiquN4YpVf2PXEQkVT
…er ocr-prep

Both modules are already pure Rust (resample uses only `image`; tf_match
only std) but were gated behind `ml`. Re-gate them to ocr-prep and make
tf_match a public module so the browser TableFormer path (phase 3) can
crop table regions (page→1024px box-average, crop→448 bilinear) and run
docling's cell matcher — the same code the native pipeline uses. Native
`ml` still pulls them in (ocr-prep ⊂ ml); the ocr-prep-only slice now
compiles both. No behavior change.

Refs docling-project#157

Signed-off-by: artiz <artem.kustikov@gmail.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EY5KAiquN4YpVf2PXEQkVT
…tail into tf_core

Extract the second half of predict_table_rows — word-to-cell matching
(docling's post-processor via tf_match, or the geometric fallback) and the
span-expanded rows×cols grid assembly — into tf_core::table_rows, shared
with the browser path. The native predict_table_rows keeps its ort head
(resample crop → predict_table_structure) and delegates the tail. The
parity debug dump (fs) stays native-only (#[cfg(feature = "ml")]) so the
shared code compiles for wasm. Pure code-motion; native output unchanged.

Refs docling-project#157

Signed-off-by: artiz <artem.kustikov@gmail.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EY5KAiquN4YpVf2PXEQkVT
…t-web interop

Wire the ONNX table-structure model into the browser pipeline. New wasm
module tableformer.rs delegates the three graphs (encoder/decoder/bbox) to
a stateful JS TfSession and drives the autoregressive loop in Rust on
tf_core: preprocess_input → encode → step* (BboxBook bookkeeping, argmax) →
bbox → merge_spans → build_table_cells → table_rows. The heavy tensors —
the encoder's constant cross K/V and enc_out, and the growing decoder
KV-cache — stay JS-side in the session, so each step marshals only the last
tag (one int) in and logits+hidden (525 floats) out, avoiding O(steps^2)
copies across the boundary (the decoder_kv graph: 6 layers, 8 KV heads,
head_dim 64, cross length 784).

ScannedConverter gains addPageTf (the lite add_page stays geometric): table
regions run predict_table_rows, others stay None, then
scanned::assemble_page_with_tables (new — the general assemble the native
pipeline uses) lays them out. TableFormer is opt-in so the 225 MB encoder
isn't fetched unless a table profile is chosen. Native ml unchanged; wasm
target + clippy clean.

Refs docling-project#157

Signed-off-by: artiz <artem.kustikov@gmail.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EY5KAiquN4YpVf2PXEQkVT
Add the browser TableFormer profile to the scan demo. A "TableFormer
tables" checkbox opts in; when checked, pipeline.js lazily loads the three
graphs same-origin from ./models/tableformer/ (encoder.onnx self-contained
~225 MB; decoder_kv.onnx + bbox.onnx with their .onnx.data sidecars via
ort-web's externalData option) and builds JsTfSession — the stateful
session the wasm TfSession interop expects: encode stashes the constant
cross K/V + enc_out and resets the KV-cache, step runs one decoder round
(int64 tag, empty first-step cache [6,1,8,0,64]) and returns logits+hidden,
bbox runs the box/class head. Table regions then go through
ScannedConverter.addPageTf; the lite geometric path stays the default so
the big models aren't fetched unless asked. Models load lazily on the
first table conversion with a progress line.

Refs docling-project#157

Signed-off-by: artiz <artem.kustikov@gmail.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EY5KAiquN4YpVf2PXEQkVT
Splitting add_page into a #[wasm_bindgen] wrapper pair plus a plain
add_page_impl left page_count and finish inside the non-wasm_bindgen impl
block, so they stopped being exported to JS ("cur.conv.finish is not a
function"). Close the plain impl right after add_page_impl and put
page_count/finish back in a #[wasm_bindgen] impl block.

Refs docling-project#157

Signed-off-by: artiz <artem.kustikov@gmail.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EY5KAiquN4YpVf2PXEQkVT
Formatting-only: rustfmt reorders the tf_core/tf_match mod declarations
and rewraps a couple of tf_core lines. Fixes the pinned fmt·clippy CI job.

Refs docling-project#157

Signed-off-by: artiz <artem.kustikov@gmail.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EY5KAiquN4YpVf2PXEQkVT
artiz and others added 10 commits July 25, 2026 09:54
Add a cross-origin layout candidate after the local ./models/ ones: a
Hugging Face mirror (pivozavrus/docling-rs-models) that sends CORS
headers, so the page works when served cross-origin (the hosted phone
demo on raw.githack) — GitHub Release assets can't be fetched that way.
Local dev still prefers ./models/.

Refs docling-project#157

Signed-off-by: artiz <artem.kustikov@gmail.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EY5KAiquN4YpVf2PXEQkVT
Diagnostic for the browser scanned path: console-log the per-page region
labels after refinement, so it's visible whether the layout flagged any
`table` region — a scanned table only fills when it does and the region's
words are OCR'd (a known gap: prep_region_lines skips non-text labels, so
table interiors currently carry no words for the matcher).

Refs docling-project#157

Signed-off-by: artiz <artem.kustikov@gmail.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EY5KAiquN4YpVf2PXEQkVT
Phones can't open the console, so route the layout region-label histogram
to the page instead: the wasm calls a global __docling_diag sink, the
worker forwards it to the main thread, and scan.html shows it in a small
#diag line under the controls (cleared on each new conversion). The
main-thread fallback defines the sink directly.

Refs docling-project#157

Signed-off-by: artiz <artem.kustikov@gmail.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EY5KAiquN4YpVf2PXEQkVT
The browser scanned path never recognized text inside table regions —
prep_region_lines skips non-text labels and there's no pdfium text layer —
so a detected table had no words for the matcher and rendered empty. Add
ocr_prep::prep_table_words: crop each table region, split into lines, then
split each line into word tokens by a vertical ink-projection profile
(segment_words), and prep each word for recognition with its own page-point
box. The browser add_page now OCRs those word crops and appends them to the
page cells; assemble routes them into the table region (geometric or
TableFormer) instead of stray paragraphs. Word-level boxes are what let the
matcher distribute a row across its columns. Native untouched (it reads
word boxes from pdfium).

Refs docling-project#157

Signed-off-by: artiz <artem.kustikov@gmail.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EY5KAiquN4YpVf2PXEQkVT
Mirror the layout-model fallback: try local ./models/tableformer/ first,
then the CORS Hugging Face base, so TableFormer works on the hosted
(phone) demo. External data is fetched from the same base that served the
.onnx.

Refs docling-project#157

Signed-off-by: artiz <artem.kustikov@gmail.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EY5KAiquN4YpVf2PXEQkVT
The models were uploaded to the repo root next to layout_heron_int8.onnx
(no subfolder), so point the HF fallback base at MODEL_BASE directly
instead of MODEL_BASE + "tableformer/".

Refs docling-project#157

Signed-off-by: artiz <artem.kustikov@gmail.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EY5KAiquN4YpVf2PXEQkVT
The load fallback swallowed the fetch error and reported a flat "not
found", hiding whether it was a 404, an HTTP 503 throttle, or an
out-of-memory on the 225 MB encoder. Keep and report the last error.

Refs docling-project#157

Signed-off-by: artiz <artem.kustikov@gmail.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EY5KAiquN4YpVf2PXEQkVT
Add a multi-file picker so the models can be selected from the device
instead of fetched: each File is read to an ArrayBuffer, handed to the
pipeline (transferred to the worker), and used by name before any network
fetch. This avoids the 380 MB TableFormer download on a phone entirely and
— since reading a File is a single allocation — sidesteps the double-
buffering peak fetchProgress hits on the 225 MB encoder (the likely cause
of the earlier load failure). Layout and TableFormer both resolve
provided-file → local ./models/ → Hugging Face, in that order.

Refs docling-project#157

Signed-off-by: artiz <artem.kustikov@gmail.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EY5KAiquN4YpVf2PXEQkVT
Rewrite the README's ML section to cover all three stages (OCR, layout,
TableFormer docling-project#157) with the model manifest, add a "Run it on your phone"
walkthrough (download_dependencies.sh from upstream master, static hosting
notes, and the device model-file picker with its device → local → Hugging
Face resolution order), and a "Performance & threading" section (coi.js
cross-origin isolation for ORT threads, the int8 layout export, and the
fp32 encoder as the dominant mobile bottleneck with a pointer to
quantize_models.py). Also wrap ocr_lines to satisfy rustfmt.

Refs docling-project#157

Signed-off-by: artiz <artem.kustikov@gmail.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EY5KAiquN4YpVf2PXEQkVT
The earlier note called an int8 TableFormer encoder "the biggest lever."
Measuring on a native x86 build over 15 real corpus table crops shows
otherwise: the encoder is a ResNet backbone feeding a 6-layer transformer,
and the transformer Gemms (not the ~20 convs) are the cost. Conv-only
static QDQ keeps output fidelity (enc_out/cross cosine >= 0.995) but is not
faster than ORT's fp32 conv kernels; quantizing the attention MatMuls
collapses cross-attention fidelity to ~0.85 (garbled structure) for no
speed gain. Encoder stays fp32; document the real levers (desktop for heavy
tables, multi-table batching) instead of a quantization that does not pay.

Refs docling-project#157

Signed-off-by: artiz <artem.kustikov@gmail.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EY5KAiquN4YpVf2PXEQkVT
@artiz
artiz force-pushed the claude/wasm-ml-stage1 branch from 065e284 to 583b706 Compare July 25, 2026 07:54
@artiz
artiz merged commit 2eeff63 into docling-project:master Jul 25, 2026
10 checks passed
@artiz
artiz deleted the claude/wasm-ml-stage1 branch July 25, 2026 08:10
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

#79 phase 2: ML pipeline in the browser (ONNX in WASM)

1 participant